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ABSTRACT 

A study was made of a noncredit home economics 
program presented by the University of Wisconsin Extension over the 
statewide Educational Telephone Network (ETN) . The study was limited 
to the total population involved in one program (54) and their 
immediate reaction to that program- Methodology used was a 
combination of action research and field research- The instrument was 
developed around six factors: environment, content, program process, 
lecturer, moderator, and technical qualities of the ETN. Variables 
concerned with the various factors were interspersed and resulted in 
48 items plus a 9-point scale for an overall assessment of the 
program. A four-point interval scale was used with each of the 48 
variables^ The in^'trument was open-ended, with provision for 
suggestions for improvement. Subjective evaluative inference based on 
means and standard deviations indicated the overall program was 
somewhat better than a possible average but with room for 
improvement. A better approach to organization and a more careful 
analysis of the level of content could lead to increased 
effectiveness. The moderator should assume the role of group leader 
in an attempt to bring about more group interaction, (author/eb) 
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EVALUATION INFERENCE APPLIED TO PROGRAMMING 

ON AN EDUCATIONAL TELEPHONE NETWORK - A BEGINNING 
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Evaluative research reported here represents one facet of a 
research responsibility designed as an integral part of a systems 
approach to utilization of instructional media resources in University 
Extension, The University of Wisconsin. Such a responsibility implied 
a broad curriculum development orientation and philosophy which con- 
sidered ^valuation more as a beginning and not an end to instructional 
planning . 

This research was conducted to discover unusual strengths and 

weaknesses in the selected areas of environment, content, program 

process, lecturer, moderator and technical qualities of the Educational 

Telephone Network as they related to a teaching-learning situation 

* 

involving an adult education class conducted via the Wisconsin 
Statewide Educational Telephone Network. 

Specific questions to be answered were: 

1. What was the populations’ reaction to the total program? 

2. What was the populations' reaction to the various factors? 

3. What variables within these factors represented unusual 
strengths and weaknesses? 

It was assumed that professional intuitive skill could be 
capitalized upon in evaluation for program improvement. It was also 
assumed that an analog could be developed which would describe ideal 
conditions in a learning situation and if that analog could be copied 
in each ETN terminal location, better ETN programming would occur. In 
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addition, it lA^as assumed that such an analog would provide a standard 
for relating pragmatic data which would give direction to program 
analysis. Evaluation was defined as a subjective use of pragmatic 
data to make evaluative inference for the purpose of developing a new 
program theory. The study was limited to total population involved 
in the one program and immediate reaction to that program was solicited. 
The idea of applying knowledge to new theories of programming was the 
real significance of the study. 

Content for the instrument was derived from literature and through 
counsel with authorities. Variables of each factor were 
interspersed resulting in 48 items, plus a nine point scale for an 
overall assessment of the program. The instrument also had a page of 
instructions, the rating criteria, and a page for gathering general 
information such as age, educational background, years of professional 
experience, and reasons for participating. Program information, such 
as date of program, name of city in which program was held, and interest 
in additional programs was also given. The instrument was open-ended 
in that it provided a place for participants to respond to the statement, 
"Suggestions for Improving this Program." A four point interval scale 
was used, thus a participant was asked to check whether an item was: 

(A) Not descriptive of the program they had just finished. 

(B) Minimal descriptive of the program just finished. 

(C) Somewhat descriptive of the program just finished. 

(d) Most descriptive of the program just finished. 

The general information and program information treatment showed 
a frequency and percentage count of responses. Population means were 
used as a basis for interpreting variable responses. 



Responses to four questions which dealt with participant 
characteristics were tabulated along with one question which dealt 
with the location and one question which inquired about interest for 
future programs. In terms of age, 40 of the 54 participants were in 
the 26 to 45 age bracket. Their educational background showed that 
49 had Bachelors degrees, four had Masters degrees, and one had a 
Ph.D.. In terms of years of experience, the majority of the parti- 
cipants had relatively few years of experience. (Fig. I) The reason 
for participating in this non credit program as indicated by the 
participants' responses was, first, for personal interest, second, for 
professional advancement. 

Participants were asked to respond to the question, "Would you be 
interested in participating in a similar program next semester?" The 
response to that question indicated 78% were interested. (Fig. II) 

Obviously this kind of information gives some insight into 
characteristics of participants in such a program. However, it is also 
quite obvious that there is need for much more precise identification of 
participant characteristics and interests. 

The total program and evaluation data (Fig. Ill) were treated 
by turning the 9 point scale from the vertical position, as it 
appeared on the instrument, to a horizontal position. A bar graph 
shows total possible response that could be made by the group, the 
actual response made by the group, and the average of the total possible 
responses. In this case the highest possible rating would have been 9, 
but the actual mean score was 5.74, thus the programmer could very 
quickly get feed back on the total project by relating the parti- 
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cipants' actual mean score with the maximum possible, scale average 
and minimum possible. 

Still in terms of stated significance of the study this did not 
give much insight into program structure. Thus each factor was 
treated the same way. (Fig. IV) The factor concerned with environ- 
ment may be read as follows: the participants' accumulated mean score 
was 24.7, the highest possible score was 28, the average of the possible 
scores was 14. Thus, one would conclude that on this program the 
environment was a strong part of the program structure; persons 
responsible for the program would want to analyze the components of 
the environment. The knowledge gained through such an analysis would 
be used when the program was again presented. 

A survey of the factor evaluation chart shows similar strength 
in the moderator factor. (Fig. IV) The content factor, lecturer 
factor, and telephone network factor show an average relationship 
between the average and possible scores; while the greatest change for 
program improvement appears to lie in the program process. This 
confirmed a belief that format was a critical factor in Instructional 
Telephone Network programming and that there were probably unique 
characteristics that need to be identified. 

In order to meet the stated significance of the study it was 
necessary to look inside the factors in an attempt to find out more 
precisely what variables had the greatest bearing on factor means. 

Thus, data for each factor was treated the same way. Analysis of the 
group mean for each variable was made by relating it to an arbitrary 
scale showing mean score categories of 2.7 or below, 3.7 or above and 



2.7 to 3.7. (Fig. V) 
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The environment factor score was influenced by two variables 
falling in the 3.7 or above category and none in the 2.7 category. 

These variables indicated that the room noise was low during the 
lecture and that the room was convenient and easy to find. Mean 
scores of the content factor were influenced by two variables falling 
in the 2.7 or below category and none in the 3.7 or above range. 

These variables indicated the content presented was not discussed at 
the appropriate level and that it did not provide enough detail. The 
program format factor score was influenced by five variables that 
fell into the 2.7 or below category. These variables indicated the 
question and answer period was long enough but not as beneficial as it 
should be. Reason for this was that organization did not give all 
participants time to ask their questions. Also, major points discussed 
were not too well outlined verbally by the lecturer before the main 
body of content. Again, there was enough time for content material but 
too little of the right kind of content presented. Variable scores 
in the remaining three factors: lecturer, moderator, and the telephone 
network, fell for the most part in the 2.7 to 3.7 category. The lecturer 
did seem to have good rapport with the participants. There seemed to 
be a need for the moderator to act more as a group leader. The ETN 
equipment seemed easy to operate. 

One would conclude that the overall program was somewhat better 
than a possible average but with room for improvement. The format of 
the program should be carefully evaluated based on the five items 
falling in the 2.7 or less category. And that such an analyses should 
result in a different format theory for the next program. The contents 
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should be analyzed especially in the areas of content level. One 
would conclude that more effort should be made to have the moderator 
act as a group leader instead of just a moderator. And that the 
characteristics of the environment should be carried over to the next 
program as should the qualities of the lecturer and the techniques 
used with ETN. 

However, it was strongly suggested that the real conclusions 
of this study were derived, not so much by the researcher, but by the 
curriculum committee as they analyzed the program based on data provided 
in preparation for the next program. 

There will be continual effort to refine the instrument both in 
terms of appropriate variables and its validity and reliability. 

There will be future research dealing with the isolation of more 
specific factors. There is need for research which will be designed 
around a hypothesis of relationship between factors and between 
variables. It is also recognized that a need for truly behavioral 
change research, as an extension of this research approach, is 
desirable. Also, a more precise means of developing program theory 
involving carefully derived experimental data will need to be secured. 
There is indeed a lot to be done! 



ERIC Cleurlnjihouse 






